{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This is a modified example based on the [MobileNetV3 Supernet NAS](https://github.com/intel/neural-compressor/blob/master/examples/notebook/dynas/MobileNetV3_Supernet_NAS.ipynb) notebook. The main goal of this notebook is to showcase distributed search functionality.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip -q install neural_compressor dynast==1.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatievely, if you have a local copy of https://github.com/intel/neural-compressor, you can uncomment and run the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import sys\n",
    "# sys.path.insert(0,'<path to neural compressor>')\n",
    "# !pip install -qr <path to neural compressor>/requirements.txt\n",
    "# !pip install -q dynast==1.1.0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "A simple script `distributed_example.py` demonstrating how to use distributed search functionality is located in the same directory as this notebook. The distributed functionality can be used with both `MPI` and `torchrun`.\n",
    "\n",
    "> Note: When run with `torchrun`, unless explicitly specified, `torch.distributed` uses `OMP_NUM_THREADS=1` ([link](https://github.com/pytorch/pytorch/commit/1c0309a9a924e34803bf7e8975f7ce88fb845131)) which may result in slow evaluation time. Good practice is to explicitly set `OMP_NUM_THREADS`  to `(total_core_count)/(num_workers)` (optional for MPI).\n",
    "\n",
    "To run distributed NAS within Neural Compressor/DyNAS-T with `MPI`/`torchrun`, please add the following line to your configuration:\n",
    "\n",
    "```python\n",
    "config.dynas.distributed = True\n",
    "```\n",
    "\n",
    "### `mpirun`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# If path to Neural Compressor was specified in the cell above, please modify the line below accordingly and uncomment it before running this cell.\n",
    "# export PYTHONPATH=<path to neural compressor>neural-compressor\n",
    "\n",
    "export PYTHONPATH=/nfs/pdx/home/mszankin/store/code/opensource/neural-compressor\n",
    "\n",
    "time mpirun \\\n",
    "    --report-bindings \\\n",
    "    -x MASTER_ADDR=127.0.0.1 \\\n",
    "    -x MASTER_PORT=1238 \\\n",
    "    -np 2 \\\n",
    "    -bind-to socket \\\n",
    "    -map-by socket \\\n",
    "        python distributed_example.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dynast_inc_pip_test",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "034f8a08a724a63543abaa4596714d81bf71b36e8b4dd0d5bf824a9fea1bc071"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
